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Abstract 


This  report  describes  the  use  of  prediction-enabled  component  technology  (PECT)  as  a  means 
of  packaging  predictable  assembly  as  a  deployable  product.  A  PECT  results  from  integrating  a 
component  technology  with  one  or  more  analysis  technologies.  Analysis  technologies  allow 
analysis  and  prediction  of  assembly-level  properties  prior  to  component  assembly,  and,  pre¬ 
sumably,  prior  to  component  acquisition.  Analysis  technologies  also  identify  required  compo¬ 
nent  properties  and  their  certifiable  descriptions.  This  report  describes  the  major  structures  of  a 
PECT.  It  then  discusses  the  means  of  validating  the  predictive  powers  of  a  PECT  so  that  con¬ 
sumers  may  obtain  measurably  bounded  trust  in  design-time  predictions.  Last,  it  demonstrates 
the  above  concepts  in  a  simple  but  illustrative  model  problem:  predicting  average  end-to-end 
latency  of  a  ‘soft’  real  time  application  built  from  off-the-shelf  software  components. 
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1  Introduction 


This  report  describes  a  prototype  prediction-enabled  component  technology  (PECT).  PECT  is 
both  a  technology  and  a  method  for  producing  instances  of  the  technology.  A  PECT  instance 
results  from  integrating  a  software  component  technology  with  one  or  more  analysis  technolo¬ 
gies.  PECT  allows  predictable  assembly  from  certifiable  components.  By  predictable  assem¬ 
bly,  we  mean: 

•  Assemblies  of  components  are  known,  by  construction,  to  be  amenable  to  one  or  more 
analysis  methods  for  predicting  their  emergent  properties. 

•  The  component  properties  that  are  required  to  make  these  predictions  are  defined,  avail¬ 
able,  and  possibly  certified  by  trusted  third  parties. 

The  underlying  premise  of  PECT  is  that  while  it  may  be  impossible  to  analyze,  and  thereby 
predict  the  runtime  behavior  of  arbitrary  designs,  it  is  possible  to  restrict  our  designs  to  a  sub¬ 
set  that  is  analyzable.  This  premise  has  already  been  seen  in  the  use  of  logical  (formal)  analy¬ 
sis  and  prediction  [Finkbeiner+01]  [Sharygina-t-Ol],  and  it  can  also  be  applied  to  empirical 
analysis  and  prediction.  It  is  a  further  premise  of  PECT  that  software  component  technology  is 
an  effective  way  of  packaging  the  design  and  implementation  restrictions  that  yield  analyzable 
designs. 

This  report  describes  and  illustrates  the  structure  of  a  PECT,  explores  the  strengths  and  limita¬ 
tions  of  this  approach  to  predictable  assembly,  and  charts  a  course  for  further  applied  research. 


1.1  Background 

This  report  presents  results  of  an  ongoing  internal  research  and  development  activity  at  the 
Software  Engineering  Institute  (SEI).  The  objective  of  this  research  is  to  accelerate  the  indus¬ 
trial  adoption  of  technologies  that  can  reliably  predict  the  runtime  behavior  of  systems.  A 
premise  of  this  research  is  that  software  component  technology  provides  an  effective  vector 
for  packaging  analysis  and  prediction  technologies.  Prediction-enabled  component  technology 
is  one  form  this  packaging  can  take. 

While  this  research  draws  upon  an  array  of  published  research  results,  here  we  describe  only 
how  this  report  is  related  to  other  SEI  reports  stemming  from  this  internal  research  effort.  Bass 
et.  al.  provide  a  market  assessment  of  component-based  software  in  [Bass-nOl].  Bachmann  et. 
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al.  discuss  the  technical  elements  of  software  component  technology  in  [Bachmann+00]. 
Wallnau  and  Stafford  outline  the  key  conceptual  distinctions  of  predictable  assembly  in  The 
Philosophy  of  Predictable  Assembly.  ^ 

1.2  Organization  of  this  Report 

Chapter  2  presents  an  overview  of  the  major  elements  of  PECT.  Chapter  3  describes  the  com¬ 
ponent  and  attribute  prediction  technologies  used  in  the  PECT  prototype,  COMTEK-X.  Chap¬ 
ter  4  describes  this  prototype  and  its  validation.  We  describe  only  enough  to  illustrate  the 
structure  of  a  PECT  and  how,  in  general,  a  PECT  must  be  validated.  In  Chapter  5,  we  summa¬ 
rize  the  key  results  and  questions  raised  by  the  prototype.  Appendix  A  and  Appendix  B  pro¬ 
vide  the  details  of  how  we  established  the  theoretical  and  empirical  validity  of  the  PECT 
prototype.  Appendix  C  shows  the  assemblies  used  in  empirical  validation. 


1.  Wallnau,  K.  &  Stafford,  J.  The  Philosophy  of  Predictable  Assembly,  (CMU/SEI-2000-TR-023), 
Pittsburgh,  PA;  Software  Engineering  Institute,  Carnegie  Mellon  University,  (to  be  published  in 
December  2001). 


2 


CMU/SEI-2001-TR-024 


2  Elements  of  PECT 


PECT  integrates  software  component  technologies  (hereafter,  ‘component  technologies’)  with 
analysis  and  prediction  technologies  (hereafter,  ‘analysis  technologies’). 

Component  technologies  impose  design  constraints  on  software  component  suppliers  and  inte¬ 
grators.  These  constraints  are  expressed  as  a  component  model  that  specifies  required  compo¬ 
nent  interfaces  and  other  development  rules  [Bachmann+00]  [Heineman+Ol].  In  today’s 
component  technology,  component  models  are  designed  to  simplify  many  aspects  of  the  inte¬ 
gration  (composition)  of  components  into  assemblies.  However,  the  stress  is  on  the  syntactic 
aspects  of  composition.  Behavioral  composition  is  not  usually  addressed,  and  where  it  is 
addressed,  it  is  usually  restricted  to  rely-guarantee  reasoning  with  pre/post-conditions  on  oper¬ 
ations.  While  rely-guarantee  reasoning  can  be  quite  useful  for  reasoning  about  correctness,  it 
is  not  particularly  useful  for  reasoning  about  other  assembly-level  properties  such  as  perfor¬ 
mance,  reliability,  and  security. 

Analysis  technologies,  for  example  for  performance  [Klein+93]  and  reliability  [Lyu+96], 
depend  on  runtime  assumptions  concerning  scheduling  policy,  process  or  thread  priority,  con¬ 
currency,  resource  management  policies,  and  many  other  factors.  A  PECT  makes  these  ana¬ 
lytic  assumptions  explicit.  We  ensure  that  a  component  technology  satisfies  these  assumptions 
through  a  demonstration  of  ‘theoretical  validity.’  We  also  ensure  that  predictions  based  on  an 
analysis  technology  are  repeatable  through  a  demonstration  of  ‘empirical  validity.’  These  vali¬ 
dations  provide  bounded  confidence  that  a  collection  of  design  constraints  on  component  sup¬ 
pliers  and  system  integrators  will  yield  systems  that  are,  by  design  and  construction, 
predictable  with  respect  to  one  or  more  critical  system  properties. 

The  structure  of  a  PECT  is  depicted  using  the  Unified  Modeling  Language  (UML)  in  Figure  1. 
As  already  noted,  a  PECT  is  an  association  of  a  component  technology  (‘Component  Technol¬ 
ogy’)  with  one  or  more  analysis  technologies  (‘Analysis  Technology’).  The  UML  association 
class  ‘Association  Validity’  stipulates  that  each  such  association  is  validated.  ‘Assumption’ 
and  ‘Interpretation’  validity,  taken  together,  demonstrate  theoretical  validity.  Each  of  these 
forms  of  validity  is  discussed  later,  and  are  elaborated  in  detail  in  the  appendices. 

In  principle,  component  technology  and  prediction  technology  can  each  be  treated  as  sepa¬ 
rately  packaged  entities — we  will  not  yet  go  so  far  as  to  call  them  “components”  in  their  own 
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right'.  Where  both  are  separately  packaged,  an  N:M  association  in  Figure  1  between  compo¬ 
nent  and  analysis  technologies  would  be  reasonable.  However,  our  perspective  is  centered  on 
component  technology,  and  how  these  can  be  extended  to  predict  emergent  (assembly-level) 
attributes.  We  also  expect  that  it  will,  in  general,  be  much  easier  to  adapt  existing  prediction 
technologies  to  work  with  existing  component  technologies  rather  than  the  converse.  Thus  the 
1:N  association  between  component  technologies  and  prediction  technologies  in  Figure  1. 


Figure  1:  PECT  Conceptual  Structure  (UML) 

2.1  Component  Technology 

While  there  is  no  iron-clad  definition  of  ‘component  technology’  any  more  than  there  is  for 
‘component,’  a  consensus  has  emerged  regarding  the  essential  elements  of  a  component  tech¬ 
nology  [Bachmann-t-00]  [Heineman-i-01]  [Szyperski-H97]; 

•  A  component  model  defines  one  or  more  required  component  interfaces,  allowable  pat¬ 
terns  of  interactions  among  components,  interactional  behaviors  among  components  and 
between  components  and  the  component  runtime,  and,  possibly,  a  programming  model  for 
component  developers. 

•  A  component  runtime  environment  provides  runtime  enforcement  of  the  component 
model.  The  runtime  plays  a  role  analogous  to  that  of  an  operating  system^  only  at  a  much 


1 .  There  is,  however,  an  interesting  anaiog  between  the  context  dependencies  that  are  included  in 
Szyperski’s  base  definition  of  software  component  [Szyperski+97],  and  the  assumptions  of  anal¬ 
ysis  technoiogies. 

2.  in  fact,  the  various  COM-based  Microsoft  component  models  are  an  integral  part  of  the  Microsoft 
operating  systems. 
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higher  level  of  abstraction,  one  that  is  usually  tailored  to  an  application  domain  or  required 
assembly  properties  (e.g.,  performance  or  security). 

•  An  assembly  environment  provides  services  for  component  development,  deployment, 
and  application  assembly.  The  assembly  environment  may  also  provide  assembly-time 
enforcement  of  the  component  model. 

Each  of  the  elements  listed  above  plays  a  role  in  PECT.  The  component  model  is  the  locus  of 
the  integration  of  component  and  analysis  technologies;  it  specifies  the  design  and  implemen¬ 
tation  constraints  that  are  required  to  enable  predictable  assembly.  The  runtime  and  assembly 
environments  are  important  insofar  as  they  enforce  at  least  some  of  these  constraints.  The  runt¬ 
ime  environment  is  itself  a  target  of  certification,  as  it  may  always  be  treated  as  a  component 
in  its  own  right,  with  properties  that  contribute  to  the  prediction  of  emergent  (assembly-level) 
attributes. 


Figure  2:  Component  Technology  and  the  Constructive  Model 


Figure  2  introduces  the  new  concept,  ‘Assembly  Model,’  as  a  refinement  of  the  traditional 
component  model.  By  this  we  suggest  that  the  assembly  model  plays  the  same  role  as  compo¬ 
nent  model,  and  may  indeed  describe  many  of  the  same  things.  There  are  two  reasons  for 
introducing  this  refinement.  First,  a  single  component  technology  may  be  restricted  or  general¬ 
ized  in  different  ways  for  different  analysis  technologies.  It  therefore  makes  sense  to  isolate 
those  changes  that  are  particular  to  a  prediction  technology.  More  fundamental,  though,  the 
refinement  allows  us  to  distinguish  between  constructive  and  analytic  interfaces. 
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The  constructive  interface  includes  those  properties  of  a  component  that  permit  it  to  interact 
with  other  components.  It  also  includes  such  things  as  the  traditional  application  programming 
interface  (API).  The  constructive  interface  corresponds  closely  to  the  typical  component 
model.  The  analytic  interface  includes  those  component  properties  that  are  required  by  an 
analysis  technology,  including  things  such  as  performance  measures,  state  transition  models, 
and  process  equations.  This  interface  does  not  correspond  to  any  existing  component  model. 

Although  Figure  2  shows  the  analytic  interface  to  be  a  constituent  of  the  assembly  model,  the 
analytic  interface  described  later  is,  in  fact,  derived  from  the  component  technology.  The  con¬ 
ceptual  model  in  Figure  2  reflects  our  assertion  that  component  models  must  be  extended  to 
include  design-time  analysis  and  prediction  of  emergent  (assembly-level)  properties. 


2.2  Analysis  Technology 

There  are  many  analysis  technologies  available  to  software  engineers.  However,  these  technol¬ 
ogies  have  not  been  developed  with  the  objective  of  being  integrated  with  software  component 
technology.  As  a  result  it  is,  in  many  cases,  difficult  to  distinguish  between  an  analysis  tech¬ 
nology  and  an  underlying  strategy  for  optimizing  a  system  with  respect  to  a  particular  (ana¬ 
lyzed)  attribute.  That  is,  analysis  technology  and  architectural  design  pattern  (sometimes 
called  a  “style”  [Shaw-i-97])  are  often  conflated.  For  example.  Simplex  is  an  architectural 
design  pattern  that  optimizes  for  fault  tolerant  system  behavior  during  replacement  of  critical 
control  functions  [Sha+95].  A  formal  definition  of  Simplex  has  been  used  to  prove  (the  stron¬ 
gest  form  of  prediction)  a  number  of  properties  [Rivera+96].  However,  the  link  between  these 
proofs  and  the  design  pattern  is  at  best  implicit,  and  there  is  no  generalization  of  these  predic¬ 
tions  over  structurally  related  patterns.  The  work  of  Klein  and  others  on  quality  attribute 
design  patterns  [KIein4'99][Bass+00]  offers  some  clues  as  to  how  analysis  models  and  their 
contingent  design  patterns  may  be  disentangled;  this  is  one  starting  point  for  PECT  research. 

In  Figure  3,  we  introduce  the  new  concept,  ‘Analytic  Model’  The  Analytic  Model  is  a  distil¬ 
late  of  an  analysis  technology.  It  defines  the  property  theory  that  underlies  the  analysis  tech¬ 
nology.  It  also  defines  the  parameters  of  this  theory,  e.g.,  the  properties  that  components  must 
possess.  For  example,  a  property  theory  that  can  be  used  to  predict  assembly-level  deadlock  or 
safety  properties  might  require  component-level  process  equations  to  describe  their  concurrent 
behavior  in  an  assembly.  Such  equations  would  be  constituents  of  the  analytic  interface  of  a 
component. 

It  is  customary  to  think  of  component  types  as  being  defined  by  one  or  more  interfaces.  In 
Enterprise  JavaBeans,  ‘SessionBean’  and  ‘EntityBean’  are  component  types  defined  by  dis¬ 
tinct  interfaces.  Since  these  interfaces  define  properties  that  govern  how  components  are  inte¬ 
grated,  we  consider  them  as  part  of  the  constructive  interface.  Naturally,  components  may 
implement  several  constructive  interfaces.  Such  components  are  therefore  polymorphic  in  that 
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they  satisfy  more  than  one  constructive  type  definition.  Components  that  satisfy  a  constructive 
interface  are  called  constructive  component.  The  analytic  model  may  also  introduce  one  or 
more  analysis-specific  component  types,  and  components  may  likewise  be  polymorphic  with 
respect  to  these  type  definitions.  Such  components  are  called  analytic  components. 


Figure  3:  Analytic  Model  and  Analytic  Interface 

An  assembly  of  constructive  components  is  called  a  constructive  assembly.  Analogously,  an 
assembly  of  analytic  components  is  an  analytic  assembly.  The  mapping  from  a  constructive 
assembly  to  an  analytic  assembly  is  called  the  analytic  interpretation  of  that  constructive 
assembly.  In  effect,  we  consider  that  the  analysis  model  defines  an  analysis-specific  view  of  an 
assembly.  The  analytic  interpretation  defines  how  these  views  are  instantiated  for  any  given 
assembly. 


2.3  Integration  Co-Refinement 

There  are  many  available  component  and  analysis  technologies  in  research  and  in  the  commer¬ 
cial  marketplace.  An  important  practical  consideration  for  our  research  is  to  demonstrate  that 
existing  technologies  can  be  integrated  into  viable  PECT  instances.  However,  since  compo¬ 
nent  and  analysis  technologies  have  developed  independently,  and  to  satisfy  different  objec¬ 
tives,  their  integration  may  not  always  be  straightforward  due  to  mismatched  assumptions. 
Where  mismatches  arise,  either  or  both  must  be  adjusted,  as  illustrated  in  Figure  4. 

The  effect  of  making  a  component  technology  more  specific  is  to  make  assemblies  more  uni¬ 
form  in  structure,  but  at  the  cost  of  further  constraining  the  freedom  of  component  developers 
and  system  assemblers.  Analogously,  making  an  analysis  technology  more  detailed  may  make 
its  predictions  more  accurate,  but  may  also  increase  the  cost  of  applying  the  technology.  Fig- 
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ure  4  depicts  three  (non-exhaustive)  alternative  ways  of  integrating  a  component  technology 
with  an  analysis  technology;  each  alternative  reflects  the  above  trade-off: 

1.  PECT-I  shows  an  integration  of  component  and  analysis  technologies  that  require  a  weak¬ 
ening  of  constraints  on  both.  The  effect  of  this  tradeoff  might  be*  to  increase  the  popula¬ 
tion  of  designs  that  admit  analysis  and  prediction,  but  at  the  cost  of  making  the  analysis 
and  hence  predictions  more  abstract  and  less  accurate. 

2.  PECT-2  shows  an  integration  where  the  component  technology  remains  unaffected  but  the 
prediction  technology  is  made  more  specific.  This  tradeoff  might  reflect  the  specialization 
of  a  prediction  technology  to  an  existing  component  technology,  or  to  the  need  for 
increased  accuracy  of  predictions. 

3.  PECT-3  shows  an  integration  where  both  technologies  are  constrained.  The  net  effect  in 
this  case  is  to  restrict  the  population  of  designs  that  admit  analysis  and  prediction  and,  pos¬ 
sibly,  to  improve  the  accuracy  of  the  resulting  predictions. 
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Technology 
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□  original  technology 
•  refined  technology 
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Figure  4:  PECT  Integration  Co-Refinement 


We  refer  to  the  integration  process  implied  by  Figure  4  as  ‘co-refinement’  since  either  or  both 
technologies  may  be  refined  (we  include  generalization  and  abstraction  in  our  admittedly  col¬ 
loquial  use  of  this  term)  to  enable  the  integration  of  component  and  analysis  technologies. 


1 .  In  genera!,  there  is  no  way  to  know  whether  a  generalization  or  restriction  of  an  analysis  technol¬ 
ogy  will  result  in  an  enlarged  or  diminished  scope,  or  enhanced  or  degraded  accuracy.  We  can 
say,  however,  that  a  restriction  on  the  component  technology  results  in  a  smaller  population  of 
allowable  designs. 
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2.4  PECT  Validation 


A  technology  that  purports  to  enable  predictable  assembly  would  be  meaningless  if  its  predic¬ 
tions  could  not  be  validated.  To  paraphrase  the  wisdom  of  Wittgenstein  for  use  in  our  own  con¬ 
text; 


A  nothing  will  do  as  well  as  a  something  (that  is,  a  prediction)  about  which 
nothing  can  be  said. 

The  consumers  of  PECT  will  want  to  know  ahead  of  time  how  much  confidence  to  place  in  the 
predictive  powers  of  the  technology.  That  is,  can  the  PECT  be  trusted') 

We  believe  that  theoretical  and  empirical  validity  must  be  established  to  engender  bounded, 
quantifiable  trust  in  a  PECT: 

•  Assumption  (theoretical)  validity  establishes  that  the  analytic  model  is  sound,  and  that  all 
of  the  assumptions  that  underlie  it  are  satisfied  either  by  the  component  technology  in  part 
or  as  a  whole,  or  by  engineering  practices  external  to  the  component  technology. 

•  Interpretation  ( theoretical )  validity  establishes  that  each  constructive  assembly  has  at  least 
one  counterpart  analytic  assembly,  and  that  if  more  than  one  such  counterpart  exists,  the 
set  of  such  counterparts  is  an  equivalence  class  with  respect  to  predictions. 

•  Empirical  validity  establishes  measurable  (and  most  likely  statistical)  evidence  of  the  reli¬ 
ability  of  predictions  made  using  the  analysis  technology.  All  analysis  technologies  must 
be  falsifiable  with  respect  to  their  predictions.  This  is  a  strong  condition  that  rules  out  a 
variety  of  “soft”  attributes  that  are  defined  using  subjective  and  non-repeatable  measures. 

All  three  forms  of  validation  are  essential,  but  we  place  special  emphasis  on  empirical  validity. 
Like  Simon,  we  accept  the  utility  of  predictive  models  even  if  their  assumptions  are  falsifiable 
with  respect  to  the  inner  workings  of  the  systems  under  scrutiny,  so  long  as  the  predictions  are 
consistently  accurate  and  useful  with  respect  to  observed  phenomena  [Simon-H96].  We  also 
observe  that  software  engineering  literature  is  notoriously  weak  with  respect  to  empirical  vali¬ 
dation  of  design  theories.  With  PECT,  we  stake  a  position  that  opposes  this  continuing  trend. 
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3  COMTEK  and  Latency  Prediction 


The  PECT  prototype  combines  the  COMTEK  component  technology  with  a  property  theory 
for  latency  prediction.  We  briefly  describe  both  as  background  to  the  PECT  prototype. 

3.1  COMTEK  Component  Technology 

COMTEK*  was  developed  by  the  SEI  for  the  U.S.  Environmental  Protection  Agency  (EPA) 
Department  of  Water  Quality.  Water  quality  analysis  is  computationally  expensive,  and  in 
many  cases,  requires  the  use  of  simulation  and  iterative  equation  solvers.  COMTEK  was  a 
proof  of  feasibility  that  third-party  simulation  components  could  be  fully  compositional,  and 
could  produce  reliable  and  scalable  water  quality  simulations. 

COMTEK  has  the  following  high-level  characteristics: 

•  It  enforces  a  typed  pipe-and-filter  architectural  style. 

A  fixed  round-robin  schedule  is  calculated  from  component  input/output  dependencies. 

•  The  execution  of  an  assembly  is  sequential,  single-threaded,  and  non-preemptive. 

•  It  runs  under  the  Microsoft  Windows  family  of  operating  systems. 

•  Components  are  packaged  and  deployed  as  Microsoft  Dynamic  Link  Libraries  (DLLs). 

Despite  its  simplicity,  the  generality  of  COMTEK  was  demonstrated  in  several  application 
domains.  The  menu  tabs  above  the  assembly  canvas  in  Figure  5  display  four  families  of  com¬ 
ponents:  Hydraulic  Interfaces,  Hydraulic  Models,  Wave  Interfaces,  and  Test  Interfaces.  Com¬ 
ponents  are  chosen  from  one  or  more  component  families,  depending  on  the  application. 

Figure  5  presents  a  screenshot  of  the  COMTEK  assembly  environment.  The  graphic  depicts  an 
assembly  built  from  components  of  the  Wave  Interface  family.  This  and  similar  assemblies  are 
the  subject  of  the  PECT  demonstration.  These  assemblies  implement  audio  signal  sampling. 


1 .  COMTEK  was  originally  called  ‘WaterBeans.’  We  have  elected  to  rename  WaterBeans  because 
our  scope  is  far  broader  than  the  domain  of  water  quality  modeling,  and  because  we  wish  to 
avoid  confusion  between  the  original  work  in  simulating  water  quality  and  our  current  work  in  pre¬ 
dictable  assembly. 


CMU/SEI-2001-TR-024 


11 


manipulation,  and  playback  functionality.  We  chose  to  develop  a  PECT  for  assembling  audio 
playback  applications  since  we  could  develop  (we  thought)  a  simple  performance  analysis 
model  to  accommodate  the  relative  simplicity  of  COMTEK. 
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Figure  5:  COMTEK  Assembly  Environment 


3.2  Predicting  the  Latency  of  COMTEK  Assemblies 

The  audio  playback  application  lies  in  the  domain  of  what  is  sometimes  referred  to  as  ‘soft 
real-time’  applications.  In  soft  real-time  applications,  timely  handling  of  events  or  other  data  is 
a  critical  element  of  the  application,  but  an  occasionally  missed  deadline  is  tolerable.  In  the 
audio  playback  application,  audio  signals  received  from  an  internal  CD  player  must  be  sam¬ 
pled  at  regular  intervals — approximately  every  46  milliseconds  for  each  1,024  bytes  of  audio 
data.  A  failure  to  sample  the  input  buffer,  or  to  feed  the  output  buffer  (i.e.,  the  speakers)  within 
this  time  interval  will  result  in  a  lost  signal.  Too  many  lost  signals  will  disrupt  the  quality  of 
the  audio  playback;  however,  a  few  lost  signals  will  not  be  noticeable  to  the  untrained  ear. 
Thus,  audio  playback  has  ‘soft’  real-time  requirements. 

The  problem  we  posed  for  PECT  was  to  predict  the  end-to-end  latency  of  an  assembly  of 
COMTEK  components,  where  latency  is  defined  as  the  time  interval  beginning  with  the  exe¬ 
cution  of  the  ‘first’  component  executed  in  an  assembly  and  ending  with  the  return  from  the 
‘last’  component  in  that  assembly  (in  a  round-robin  schedule,  the  notions  of  ‘first’  and  ‘last’ 


12 


CMU/SEI-2001-TR-024 


are  relative,  but  we  assume  there  is  some  designated  ‘first’  and  ‘last’).  This  will  allow  engi¬ 
neers  to  predict  whether  a  particular  assembly  will  satisfy  its  performance  requirements  prior 
to  its  integration,  and  possibly,  prior  to  acquiring  the  components.  Such  predictions  must  be 
made  despite  the  fact  that  the  Windows  platforms  we  used  make  no  performance  guarantees. 

Our  emphasis  in  this  work  was  on  understanding  PECT  rather  than  in  solving  the  latency  pre¬ 
diction  problem.  We  were  hoping  to  achieve  prediction  to  within  10%  of  observed  assembly 
latency — good  enough  to  demonstrate  the  PECT  concept,  even  if  insufficient  for  real  engi¬ 
neering  practice. 

As  will  be  seen,  however,  we  did  much  better  than  a  10%  margin  of  error,  although  this  was 
never  a  goal  of  the  prototype. 
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4  Illustration 


This  chapter  describes  the  packaging  of  latency  analysis  and  prediction  with  COMTEK.  We 
refer  to  the  resulting  integration  as  COMTEK-A,.* 


4.1  Assembly  Model 

As  explained  earlier,  an  assembly  model  defines  the  set  of  (constructive  and  analytic)  compo¬ 
nent  types  recognized  by  a  component  technology,  and  also  specifies  rules  for  their  composi¬ 
tion  into  assemblies  that  can  be  analyzed  with  one  or  more  analytic  models.  A  specification  of 
COMTEK  can  be  found  in  [Plakosh+99].  This  specifies,  among  other  things,  the  interface  that 
components  must  implement  the  types  of  properties  allowed,  the  data  structures  used  to  sup¬ 
port  introspection,  and  the  ways  to  transfer  data  between  components. 

We  adapted  the  original  COMTEK  specification  to  reflect  the  additional  requirements  of 
latency  analysis  of  COMTEK-X,  and  to  explore  how  the  requisite  aspects  of  the  constructive 
and  analytic  interfaces  might  be  specified.  Figures  6, 7,  and  8  show  UML  specifications  of 
three  views  of  the  assembly  model.  In  these  views  we  use  UML  stereotypes  to  distinguish  the 
constructive  (‘<<constructive»’)  and  analytic  (‘<<analytic»’)  interfaces. 

At  this  stage  of  our  research,  we  are  uncertain  about  the  best  way  to  document  the  assembly 
model,  and  indeed  are  uncertain  about  precisely  what  lies  within  the  scope  of  the  assembly 
model.  What  follows  should  be  interpreted  as  suggestive  rather  than  normative.  The  three 
views  of  the  assembly  model  illustrated  below  define  the  component  metatype,  interaction 
rules,  and  dynamic  behavior  of  COMTEK-A,  assemblies,  respectively. 


4.1.1  Component  Metatype  Specification 

The  component  metatype  defines  interfaces  and  other  rules  that  components  must  satisfy,  that 
is,  what  it  means  to  be  a  COMTEK  component.  This  includes  interface,  pre-  and  post-condi- 


1 .  We  modify  the  name  of  the  base  component  technology  with  an  attribute  designator  that  Indi¬ 
cates  the  types  of  analyses  that  are  enabled  by  the  PECT.  The  form  of  a  PECT  name  is  as  fol¬ 
lows: 

component  technology  designator[-attribute  theory  designator]'*’ 

More  than  one  attribute  theory  might  be  used  with  the  same  component  technology  simulta¬ 
neously,  and  thus  a  string  of  one  or  more  attribute  theory  designators  is  used.  We  use  greek  let¬ 
ters  to  designate  attribute  theories,  and  X  for  the  latency  theory. 
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tions,  invariants,  and  packaging.  Note  that  we  include  packaging  in  this  list  because  this  speci¬ 
fication  must  describe  the  component  types  as  deployable  units.  Therefore,  the  specification  of 
the  binary  form  a  component  must  take,  for  example  a  dynamic  link  library  (DLL)  or  a  Java 
archive  (JAR),  is  also  part  of  the  constructive  model. 

Figure  6  shows  a  UML  model  of  the  metatype  for  COMTEK-X  components.  Many  things  have 
been  abstracted  away  in  this  model — but  it  gives  an  example  of  the  things  that  have  to  be  spec¬ 
ified,  such  as  required  properties,  methods,  and  invariants.  Instances  of  ‘ComponentType’  are 
deployable  units  that  are  component  factories — they  provide  runtime  instances  of  themselves 
via  their  ‘getNewInstance  { )  ’  method.  For  example,  let  us  assume  that  we  have  an 
instance  of  ‘ComponentType’,  whose  ‘componentTypeName’  property  is  set  to  ‘Wave View.’ 
Then,  ‘WaveView’  is  a  component  that  can  be  deployed,  and  can  provide  instances  of  itself  via 
‘getNewInstance  ( ) .’ 


Figure  6:  COMTEK-X  Component  Metatype 


More  significant  for  our  purposes  is  the  differentiation  of  constructive  and  analytic  interfaces. 
Input  and  output  ports  and  a  set  of  properties  constitute  the  constructive  interface.  The  analytic 
interface  consists  of  two  mandatory  analytic  properties,  ‘p’  and  ‘e,’  whose  meanings  are 
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described  later  in  this  report.  Note  that  UML  does  not  permit  co-habitation  of  classes  and 
instances  in  models.  Thus,  although  ‘p’  and  ‘e’  are,  logically,  instances  of  ‘Property,’  they  are 
depicted  as  classes  in  Figure  6.  Also  associated  with  ‘p’  and  ‘e’  properties  are  constraints  on 
their  values,  specified  in  the  UML  Object  Constraint  Language  (OCL). 

4.1.2  Interaction  Rules 

The  interaction  model  describes  rules  for  composing  components  into  assemblies.  Figure  7 
shows  some  aspects  of  the  COMTEK-X  assembly  model.  This  model  defines  the  way  that 
components  can  be  connected,  and  specifies  invariants  for  the  whole  assembly.  The  ‘Compo¬ 
nent’  class  in  the  diagram  represents  an  instance  of  ‘ComponentType’  from  Figure  6.  The 
association  class  ‘Connection’  represents  how  components  are  linked  together.  In  addition, 
OCL  invariants  represent  the  constraints  that  must  be  observed  when  composing  components; 
for  example,  a  component  cannot  connect  to  itself,  and  the  data  types  of  the  output  and  input 
ports  must  be  the  same.  In  the  case  of  COMTEK-X  applications,  we  are  also  interested  in  a 
property  of  the  application  that  indicates  whether  an  assembly  is  fully  connected,  because  the 
COMTEK-X  runtime  environment  will  not  execute  the  application  unless  it  is  fully  connected. 
The  definition  of  ‘fullyConnected’  for  component  and  application  are  defined  in  OCL. 


Figure  7:  COMTEK-X  Interaction  Rules 
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4.1.3  Assembly  Behavior 


The  assembly  model  may  also  detail  runtime  aspects  of  the  component  technology,  such  as 
when  and  how  instances  are  created  and  initialized,  scheduling,  and  data  transferred  among 
components.  This  information  may  be  vital  not  only  for  implementing  the  component  runtime 
environment,  but  also  for  constructing  a  PECT.  For  latency  prediction,  there  are  several  ques¬ 
tions  we  need  to  answer: 

•  How  are  components  scheduled? 

•  Can  components  be  preempted? 

•  Do  components  block  on  resources? 

•  Can  components  have  different  priorities? 

•  Can  components  be  multi-threaded? 

The  above  list  is  not  exhaustive  and  will,  of  course,  vary  from  analysis  model  to  analysis 
model.  This  kind  of  information  is  also  important  for  component  developers  and  application 
assemblers.  For  instance,  a  component  technology  that  executes  components  in  a  multi¬ 
threaded  environment  might  require  the  component  developer  to  synchronize  accesses  to 
shared  resources. 


Figure  8:  Assembly  Behavior  and  Definition  of  Assembly  Latency 
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Figure  8  shows  a  runtime  view  of  the  assembly  model.  In  this  model,  ‘A’  is  an  assembly  of 
components;  ‘p(c)’,  where  c  is  a  component  such  that  c  G  A,  is  the  number  of  times  c  has  been 
executed;  ‘getTime  ( )  ’  is  a  function  that  returns  the  current  value  of  the  system  clock.  The 
model  shows  that  components  are  executed  once  per  cycle.  There  are  other  runtime  details  that 
we  have  abstracted,  such  as  the  order  in  which  components  are  executed.  Execution  order  is 
not  required  to  predict  end-to-end  latency  of  a  COMTEK-X  assembly  in  steady  state  (as  will 
be  explained  in  the  appendix).  However,  this  information  might  be  needed  in  other  circum¬ 
stances. 

Also  defined  in  Figure  8  are  the  runtime  interpretations  of  assembly  latency  (‘A.latency’)  and 
component  latency  (cj. latency).  These  definitions  serve  two  purposes.  First,  they  give  (reason¬ 
ably)  unambiguous  model  definitions  of  latency  in  the  context  of  a  particular  component  tech¬ 
nology.  Second,  they  describe  how  the  properties  will  be  measured  for  empirical  validation. 


4.2  Analytic  Model 

The  analytic  model  defines  a  property  theory  used  to  predict  the  latency  of  an  assembly  of 
components  based  on  their  measured  properties.  In  addition,  the  analytic  model  also  exposes 
the  assumptions  that  must  be  satisfied  for  the  analysis  to  be  valid. 

4.2.1  The  Property  Theory 

The  COMTEK-X  latency  theory,  denoted  as  A^^,  is  summarized  by  the  following  equation: 

A.latency  =  maxH  ^  4)j.e+  ^  ©j.ej,  max({Oj  G  A|  Oj.p})^  Eq.  1 


We  denote  an  assembly  as  the  set  of  components  A,  and  the  component  of  A  is  denoted  as 
either  or  ©j.  or  ^>15.  These  correspond  to  one  of  two  analytic  component  types:  0  refers  to  com¬ 
ponents  that  only  have  dependencies  that  are  internal  to  A,  while  refers  to  components  that 
also  exhibit  dependencies  on  external  periodic  events  (these  symbols  were  chosen  for  their 
mnemonic  value).  A  property  of  a  component  or  assembly  is  denoted  using  ‘dot’  notation. 

A.  latency  is  the  end-to-end  latency  of  an  assembly.  Each  component  has  two  required  prop¬ 
erties  that  describe  its  latency  information;  ‘<I).e’  and  ‘O.p,’  while  each  0  component  has  only 
the  required  property  ©.e,  where  e  and  p  are  defined  as  (also,  refer  to  Figure  6): 

•  e\  is  the  execution  time  of  a  component,  exclusive  of  component  blocking  time. 

•  p:  is  the  period  of  the  external  event  on  which  a  4>  depends  and  may  block. 
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The  function  max  returns  the  largest  of  its  arguments.  Note  that  this  analytic  model  is  not 
parameterized  by  invocation  order  or  connections  among  components.  Neither  the  summation 
nor  max  depend  on  the  order  of  the  components  in  A  in  Eq.  1. 

A  more  precise  specification  of  the  COMTEK-A,  component  metatype  in  Figure  6  would  have 
shown  0  and  O  as  subtypes  (UML  subclasses)  of  ‘ComponentType.’  We  have  not  done  so 
only  to  avoid  over-complicating  the  presentation.  It  is  also  worth  noting  that  these  component 
types  only  have  meaning  within  an  A/v  interpretation,  i.e.,  within  an  analytic  assembly. 


4.2.2  Adapting  COMTEK  for  Latency  Prediction 

For  reasons  of  expediency,  we  chose  to  derive  Ay^  from  COMTEK  rather  than  begin  with  a 
more  robust  performance  theory  such  as  rate  monotonic  analysis  (RMA).  One  of  our  goals  was 
to  modify  COMTEK  as  little  as  possible,  and  derivation  was  a  straightforward  way  of  achiev¬ 
ing  this  goal.  We  were  not  interested  in  re-engineering  or  extending  COMTEK  to  address  a 
realistic  spectrum  of  soft  real-time  analysis  issues. 

Nonetheless,  we  found  it  necessary  to  impose  new  constraints  on  the  use  of  COMTEK  to  sup¬ 
port  latency  prediction.  These  constraints  are  above  and  beyond  those  that  are  imposed  by 
COMTEK  itself.  In  particular: 

1.  The  value  of  ‘e’  for  all  components  is  constant  over  all  executions  of  that  component. 

2.  The  value  of  ‘p’  is  likewise  constant  for  each  external  periodic  event. 

One  question  posed  by  these  constraints  is  whether  they  should  be  documented  as  part  of  the 
constructive  interface  or  analytic  interface.  On  the  one  hand,  they  are  meaningful  only  within 
the  context  of  A/^,  and  are  therefore  arguably  part  of  the  analytic  interface  of  PECT-A.  On  the 
other  hand,  it  is  conceivable  that  such  constraints  might  be  enforced  by  a  component  technol¬ 
ogy,  which  would  argue  that  they  belong  to  the  constructive  interface. 

In  this  illustration,  neither  constraint  can  easily  be  enforced  by  the  design  or  runtime  environ¬ 
ment;  they  must  be  enforced  by  engineering  processes  such  as  code  inspection.  We  therefore 
assign  them  to  the  analytic  interface.  This  is,  perhaps,  a  minor  point  whose  resolution  will  sur¬ 
face  with  greater  experience  with  PECT. 


4.3  Association  Vaiidity 

PECT  will  be  of  little  value  unless  its  users  (application  designers  and  engineers)  trust  the  pre¬ 
dictions.  Trust  is  a  complex  social  phenomena  that  balances  many  factors,  only  some  of  which 
can  be  addressed  by  technology.  Nonetheless,  we  must  provide  a  technical  foundation  for  trust 
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so  that  other  non-technical  factors  can  be  addressed.  We  provide  the  technical  foundation  for 
trust  in  a  PECT  prediction  by  validating  the  associations  between  a  component  technology  and 
one  or  more  prediction  technologies.  We  referred  to  this  in  Figure  1  as  association  validity. 

Association  validity  takes  two  forms;  theoretical  and  empirical  validity.  Theoretical  validity  is 
concerned  with  the  soundness  of  the  analysis  model  and  the  way  this  model  is  integrated  with 
a  component  technology.  Empirical  validity  is  concerned  with  the  reliability  and  accuracy  of 
the  predictions  made  using  the  PECT.  The  following  discussion  provides  a  thumbnail  sketch 
of  how  association  validity  was  established  for  COMTEK-A,.  A  detailed  exposition  of  theoret¬ 
ical  validity  is  provided  in  Appendix  A,  while  Appendix  B  is  concerned  with  the  details  of 
empirical  validity. 


4.3.1  Theoretical  Validity 

How  does  one  go  about  establishing  the  validity  of  a  scientific  theory?  In  an  important  sense, 
the  ultimate  arbiter  of  theoretical  validity  lies  in  predicting  phenomena  that  are  observed  under 
experimental  conditions;  this  validation  corresponds  to  what  we  refer  to  as  empirical  validity. 

Often,  though,  empirical  validity  is  not  of  itself  sufficient,  and  it  is  usually  not  the  starting 
point  for  establishing  the  validity  of  a  theory.  This  is  certainly  true  of  the  established  physical 
sciences,  where  a  theory  will  undergo  extensive  scrutiny  before  it  is  tested  experimentally.  We 
believe  this  should  also  true  of  the  theories  underlying  software  engineering  practice. 

There  appears  to  be  two  key  questions  that  must  be  asked  prior  to  investing  the  time  and  effort 
required  to  empirically  validate  a  PECT:' 

1 .  Is  the  property  theory  sound? 

2.  Can  the  theory  be  falsified? 

We  consider  the  first  question  to  lie  in  the  province  of  theoretical  validation,  while  the  second 
question  lies  in  the  province  of  empirical  validation,  although  in  the  strict  sense  no  theory  can 
be  validated  but  can  rather  only  be  falsified. 

•  Assumption  validity  establishes  that  any  mathematics  used  to  describe  the  theory  are 

sound,  and  that  if  the  theory  purports  to  describe  causality  (not  strictly  necessary  in  a  prop¬ 
erty  theory),  there  is  a  clear  link  between  theory  elements  and  mechanisms  in  the  underly¬ 
ing  software  system. 


1 .  This  may  well  be  true  in  the  physical  sciences,  too,  although  we  make  no  strong  claims  on  this 
matter. 
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•  Interpretation  validity  establishes  that  each  constructive  assembly  has  an  interpretation  in 
the  theory.  This  validation  addresses  the  second  two  key  questions  by  demonstrating  the 
relationship  between  constructive  assemblies  and  model  theoretic  assemblies  (question 
two),  and  by  providing  a  basis  for  theory  falsification  (question  three). 

We  briefly  describe  how  we  demonstrated  these  forms  of  validity  before  turning  to  the  ques¬ 
tion  of  empirical  validation. 


Assumption  Validity 

As  we  noted  earlier,  was  derived  from  COMTEK.  That  is,  the  latency  theory  A/^  emerged 
from  a  detailed  understanding  of  the  mechanisms  that  manifest  the  property — the  COMTEK 
runtime  environment.  Demonstrating  association  validity  therefore  reduced  to  demonstrating 
the  validity  of  this  derivation.  A  demonstration  of  association  validity  has  the  dual  effect  of 
demonstrating  the  mathematical  soundness  of  the  derivation,  and  highlighting  those  qualities 
of  COMTEK  that  A^  depends  upon,  i.e.,  its  assumptions. 

To  demonstrate  COMTEK-A  association  validity,  we  began  by  itemizing  the  assumptions  that 
A/^  would  likely  depend  upon,  and  by  assigning  names  to  and  carefully  defining  these  assump¬ 
tions.  These  assumptions  included  scheduling  policy,  concurrency  policy,  how  components 
and  assemblies  are  defined,  what  it  means  for  an  assembly  to  be  in  steady  state,  and  so  forth. 
From  these  definitions  we  proceeded  in  stepwise  fashion  to  derive  A^.  The  entire  derivation 
takes  the  form  of  a  sequential  argument,  with  the  dependencies  between  later  steps  to  earlier 
steps  explicitly  noted. 

Of  itself,  assumption  validity  is  not  particularly  useful  for  generating  trust.  In  fact,  our  first 
latency  theory  was  also  derived  from  COMTEK,  but  proved  to  be  inadequate  during  empirical 
validation.  We  had,  in  effect,  missed  a  key  phenomenon  of  COMTEK  assemblies:  that  compo¬ 
nents  might  block  on  periodic  events  external  to  COMTEK  and  its  assemblies.  However, 
assumption  validity  can  significantly  enhance  trust  when  used  in  conjunction  with  empirical 
validation.  In  that  situation,  effective  predictions  are  combined  with  an  explanation  of  why  the 
prediction  theory  holds.  (This  is  true  in  natural  science,  too.) 


Interpretation  Validity 

Recall  that  an  analytic  assembly  is  an  interpretation,  or  mapping,  of  a  constructive  assembly 
under  some  property  theory.  For  COMTEK-X,  interpretation  validity  results  from  demonstrat¬ 
ing  that  this  mapping  is  both  complete  and  consistent: 

•  By  complete,  we  mean  that  all  constructive  assemblies  can  be  interpreted  under  the  prop¬ 
erty  theory.  Note  that  this  does  not  refer  to  the  completeness  of  a  property  theory  with 
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respect  to  its  assumptions  about  a  component  technology.  It  is  the  task  of  empirical  valida¬ 
tion  to  ferret  out  such  missing  assumptions. 

•  By  consistent,  we  mean  that  all  interpretations  of  a  particular  constructive  assembly  will 
result  in  the  same  prediction.  Consistency  is  only  an  issue  if  there  are  several  valid  inter¬ 
pretations  of  a  constructive  assembly  under  a  property  theory.  Consistency  means  that  all 
such  interpretations  form  an  equivalence  class  with  respect  to  the  property  theory. 

Demonstrating  the  completeness  of  A/^  was  trivial,  since  there  was  a  complete  mapping  of 
constructive  component  types  to  analytic  component  types,  and  since  interactions  among  con¬ 
structive  components  were  not  parameters  of  (see  Eq.  1  in  Section  4.2.1  on  page  19),  and 
hence  need  not  be  considered.  Demonstrating  the  consistency  of  was  likewise  trivial.  How¬ 
ever,  had  A/^  not  been  specialized  to  deal  only  with  steady  state  latency,  execution  order  would 
have  been  significant  (in  non  steady-state).  In  this  case,  demonstrating  completeness  and  con¬ 
sistency  would  have  been  slightly  more  involved. 

4.3.2  Empirical  Validity 

Establishing  empirical  validity  consists  of  demonstrating  that  the  predictions  made  using  an 
analytic  model  conform  to  observations.  Thus,  our  confidence  in  the  quality  of  a  PECT  is  lim¬ 
ited  by  the  stability  of  measured  assembly-level  properties  and  their  comparisons  to  predicted 
assembly-level  properties.  Our  confidence  in  predictions  is  also  bounded  by  our  confidence  in 
the  measurements  of  component  properties  that  parameterize  property  theories.  It  should  not 
be  surprising,  then,  that  empirical  validity  rests  on  a  foundation  of  measures  and  measurement. 
Nor  should  it  be  surprising  that  statistical  analysis  plays  an  important  role  in  establishing 
empirical  validity.  All  measurement  processes  introduce  error,  and  the  abstraction  of  complex 
phenomena  into  property  theories  invariably  introduces  additional  error.  We  must  use  statistics 
to  quantify  both  forms  of  error. 

Empirical  Validation  and  Formal  Property  Theories 

Before  continuing  with  a  discussion  of  the  use  of  statistics  in  establishing  the  empirical  valid¬ 
ity  of  COMTEK-X,  we  digress  to  discuss  the  relevance  of  empirical  validation  of  formal  prop¬ 
erty  theories.  We  might  argue  that  there  is  little  (if  any)  use  in  empirically  validating  theories 
that  are  established  by  proof  theoretic  means.  To  examine  this  argument,  we  focus  only  on  the 
proof  of  component  properties.  We  consider  the  case  where  a  particular  component  behavior 
has  been  specified,  and  we  must  establish  that  the  implementation  conforms  to,  or  satisfies, 
this  specification,  using  model  checking  [Clarke+99]. 

Suppose  a  model  checking  proof  of  satisfaction  is  constructed.  This  would  be  sufficient  only  if 
we  had  a  similar  proof  that  the  component  environment  possesses  and  satisfies  its  own  formal 
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specification.  This  argument  will  regress  from  the  dependencies  of  that  environment  to  some 
other  environment,  etc.,  but  it  will  not  regress  forever.  The  ultimate  environment  is  provided 
by  a  physical  device  such  as  a  microprocessor.  Notwithstanding  formal  verification  of  micro¬ 
processor  logic,  all  physical  devices  are  subject  to  manufacturing  defects  and  wear  that  can 
only  be  detected  by  empirical  means.  Thus,  all  proof  demonstrations  ultimately  rely  upon 
empirical  demonstration. 

This  reasoning  carries  a  whiff  (or  a  strong  odor?)  of  sophistry.  Yet  it  does  demonstrate  the  ten¬ 
uous  nature  of  formal  demonstrations  of  software  behavior.  As  observed  by  Messerschmitt  and 
Szyperski,  the  boundary  between  software  and  hardware  is  virtually  non-existent,  as  software 
can  always  simulate  hardware,  and  hardware  can  always  realize  software  [Messersch+01];  the 
choice  of  which  to  use  (hardware  or  software)  is  an  economic  rather  than  theoretical  question. 
From  this  perspective  it  may  be  more  reasonable  to  consider  the  selection  of  logical  (software) 
or  empirical  (hardware)  theories  likewise  to  be  a  matter  of  practicality. 

On  a  practical  note,  proving  satisfaction  is  difficult  and  costly,  and  researchers  are  investigat¬ 
ing  how  to  use  empirical  methods  to  obtain  a  statistical  proof  of  satisfaction  of  a  formal  speci¬ 
fication  [Giannak+01].  Such  approaches  rely  upon  comparing  traces  of  program  execution  to 
traces  produced  by  symbolic  execution  of  a  specification.  In  this  way,  demonstrating  satisfac¬ 
tion  is  reduced  to  demonstrating  complete  test  coverage,  or  some  statistical  percentage  of  cov¬ 
erage. 

We  do  not  argue  against  the  utility  of  formal  property  theories.  It  is  clear  that  where  logical 
property  theories  exist  and  can  be  practically  used,  the  burden  of  empirical  validation  may  be 
significantly  reduced.  We  are  merely  suggesting  that  empirical  validation  can  probably  never 
be  eliminated  in  practice,  even  if  software  analysis  and  prediction  becomes  fully  formal.  Still, 
it  remains  a  question  for  further  research  to  demonstrate  a  seamless  way  of  integrating  and 
packaging  a  mix  of  formal  and  empirical  property  theories. 

Statistical  Approach  to  Validating  COMTEK-?i 

The  use  of  statistics  in  empirical  validation  allows  us  to  infer  characteristics  of  the  property 
theory  from  samples  of  its  application.  Confidence  intervals  and  significance  levels  provide 
more  insight  in  the  analytic  model,  so  that  an  application  assembler  can  interpret  the  results  of 
the  analytic  model,  or  compare  the  results  from  competing  models.  The  number  of  samples 
that  are  used  for  empirical  validation  depends  on  how  difficult  or  expensive  it  is  to  get  a  data 
point  for  the  sample.  This,  in  turn,  depends  on  the  kind  of  property  that  is  being  predicted. 
Therefore,  it  is  expected  that  there  will  be  tradeoffs  between  the  accuracy  of  empirical  and  sta¬ 
tistical  analysis,  and  the  cost  of  producing  this  analysis.  This  tradeoff  must  be  quantified. 
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We  use  the  average  of  a  set  of  measurements  to  estimate  the  population’s  mean.  By  using  an 
estimator,  we  incur  an  error  that  we  want  to  quantify.  To  do  so,  we  define  a  confidence  interval 
for  the  estimation,  as  follows: 


X  -  ta/2-p  <  It  <  X  +  t„/2-^ 

Vn  vn 


Eq.2 


where  fX  is  the  mean  of  the  population;  x  and  s  are  respectively  the  average  and  the  standard 
deviation  of  the  sample;  n  is  the  size  of  the  sample;  and  i^/2  is  the  t-value  from  a  t-distribu- 
tion  with  v  =  n  -  1  degrees  of  freedom  leaving  a  tail  with  an  area  of  tx/2  to  the  right  [Wal- 
pole+89].  With  this  formula,  we  calculate  a  (1  -  a)100%  confidence  interval  for  the 
population’s  mean.  In  other  words,  if  a  =  0.05,  we  can  be  95%  confident  that  |X  falls  within 
that  interval.  This  calculation  assumes  that  the  sample  comes  from  a  normal  population  (i.e.,  a 
population  that  has  a  normal  distribution).  When  the  sample  is  large  (n  >  30),  we  can  use  the 
following  formula  even  when  normality  cannot  be  assumed: 

X-Za/2-p<^<X  +  Z„/2-p  Eq.  3 

vn  Vn 


where  z^/2  the  z-value  from  a  normal  distribution  that  leaves  a  tail  with  an  area  of  cx/2  to 
the  right.  When  the  sample  is  not  large  and  we  cannot  be  sure  about  the  normality  of  the  popu¬ 
lation,  there  are  some  procedures  that  can  be  applied  both  to  test  for  normality  and  to  normal¬ 
ize  the  sample.  An  example  of  this  can  be  found  in  Appendix  B. 

The  core  of  establishing  empirical  validity  is  to  demonstrate  how  accurate  and  repeatable  the 
predictions  are  on  average.  Two  methods  we  can  use  to  do  so  are  magnitude  of  relative  error 
(MRE)  and  correlation  analysis.  These  methods  have  been  used  in  the  empirical  validation  of 
other  software  related  estimation  models  [Kemerer-i-87]. 

MRE  is  a  measure  of  the  percentage  error  of  the  predicted  property,  and  it  is  defined  as  fol¬ 
lows: 


MRE= 


A.p-A.p^ 

A.p' 


Eq.4 


where  A.p  is  the  predicted  property  and  A.p'  is  the  measured  property.  If  we  used  absolute 
error,  it  would  be  difficult  to  compare  the  errors  for  different  predictions.  For  example,  an 
error  of  0.3  would  be  relatively  small  for  an  actual  value  of  140  while  it  would  be  very  large 
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for  an  actual  value  of  1 .44.  Therefore,  we  want  to  use  relative  error  instead  of  absolute  error  so 
that  we  can  compare  and  analyze  the  predictions  for  different  assemblies.  As  we  are  concerned 
with  the  goodness  of  the  model  on  average,  we  can  compute  the  predictions  for  a  sample  of 
assemblies  and  use  the  average  MRE  to  estimate  the  mean  of  the  MREs  for  all  possible  assem¬ 
blies.  Again,  following  the  same  rationale  and  procedure  as  before,  we  can  calculate  a  confi¬ 
dence  interval  for  this  estimation. 

The  second  method  we  can  use  for  evaluating  the  goodness  of  a  model  is  correlation  analysis. 
This  method  measures  the  strength  of  the  linear  association  between  two  variables  [Wal- 
pole+89].  The  sample  correlation  coefficient  R  is  calculated  from  a  sample  of  observed  pairs 
of  values  for  the  two  variables  X  and  Y.  The  former  is  called  the  independent  variable,  and  the 
latter  is  called  the  dependent  variable.  For  our  purposes,  we  can  use  the  predicted  property  as 
the  independent  variable,  and  the  actual  property  as  the  dependent  variable. 

In  practice,  the  sample  coefficient  of  determination  is  used  to  evaluate  the  correlation.  A 
value  of  =  1  means  that  there  is  a  perfect  correlation  between  X  and  Y,  (i.e.,  between  the  pre¬ 
dictions  and  the  actual  property  values),  whereas  an  =  0  indicates  that  there  is  no  correla¬ 
tion  at  all.  In  general,  we  can  think  of  R^  as  an  indication  of  the  amount  of  variability  of  T  that 
is  accounted  for  by  its  linear  relation  with  X.  For  instance,  R^  =  0.9  indicates  that  the  model 
explains  90%  of  the  actual  value  of  the  property. 

When  correlation  analysis  is  used  for  empirical  validation,  it  is  also  important  to  determine  the 
significance  of  the  correlation  so  that  it  can  be  shown  that  the  linear  relation  in  the  sample  did 
not  happen  by  chance.  This  can  be  done  by  using  the  critical  values  for  R,  which  can  be  found 
on  tables  or  with  computer  software  based  on  the  size  of  the  sample  and  the  desired  signifi¬ 
cance  level.  If  R  is  greater  than  the  critical  value,  then  the  correlation  is  considered  statistically 
significant. 

Statistical  Validation  of  COMTEK-X 

A  summary  of  the  statistical  validity  of  COMTEK-X  is  shown  in  Table  1.  The  MRE  leads  to  a 
confidence  interval  that  suggests  that  latency  predictions  are,  on  average,  quite  accurate.  That 
is,  if  we  create  several  sets  of  random  assemblies,  the  average  MRE  of  each  set  will  fall  within 
the  confidence  interval  for  95%  of  the  sets.  The  high  R“  is  also  a  strong  indicator  of  the 
strength  of  correlation  between  observed  data  and  predictions. 

From  the  empirical  (statistical)  validation,  it  would  seem  that  COMTEK-X  offers  a  promising 
solution  to  predicting  latency  of  soft-realtime  applications.  But  we  are  right  to  be  skeptical  and 
recall  the  old  adage:  “statistics  don’t  lie,  but  liars  use  statistics.”  We  offer  the  following  cave¬ 
ats  not  in  the  mode  of  self  confession,  but  rather  to  pose  the  question:  how  should  a  PECT  be 
packaged  to  expose  the  limitations  of  an  analysis  model  and  its  validation? 
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Caveats 


The  critical  eye  can,  without  too  much  effort,  spot  a  variety  of  weaknesses  of  the  COMTEK-^ 
latency  theory  and  its  empirical  validation. 


Statistical  Analysis 

Latency  Theory 

MRE 

0.554% 

95%  Confidence 
Interval 

Lower:  0.010% 
Upper:  0.701% 

0.9999 

Significance  Level 

0.01 

MRE:  measure  of  relative  error 

R^:  coefficient  of  determination 

Table  1:  Empirical  Validation  of  COMTEK-k 

First,  how  do  we  know  that  the  assemblies  used  in  empirical  evaluation  (see  Appendix  C)  rep¬ 
resent  the  set  of  all  possible  COMTEK  assemblies?  An  examination  will  show  that  they  are 
quite  similar  in  the  number  of  components,  component  interactions,  and  component  latencies. 
A  thorough  empirical  validation  would  characterize  COMTEK  assemblies.  For  example,  the 
assemblies  might  be  characterized  by  the  number  of  components,  their  average  latency,  the 
number  or  ratio  of  input/output  connections,  and  so  forth.  Extreme  values  for  these  parame¬ 
ters,  and  parameter  combinations,  could  be  used  to  describe  boundary  cases  with  which  to  test 
empirical  validity.  The  validity  of  the  latency  theory  might  well  vary  across  these  extremes. 

Second,  we  did  not  parameterize  the  latency  theory  in  Eq.  1  with  a  variety  of  environmental 
factors  such  as  processor  and  memory  speed,  amount  of  free  memory,  resource  contention,  or 
other  factors  that  may  strongly  influence  latency.  There  is  no  doubt  that  we  could  extend  the 
latency  theory  with  such  factors,  and  we  may  do  so  to  incorporate  sensitivity  analysis  and 
multi-regression  analysis,  and  to  further  explore  packaging  PECTs  and  PECT-conformant 
components.  While  this  serious  deficiency  in  the  empirical  evaluation  should  be  made  explicit 
in  the  packaging  of  a  PECT,  the  question  remains:  can  any  standard  approach  to  PECT  label¬ 
ing  reliably  identify  such  limitations,  whether  they  result  from  commission  or  merely  omis¬ 
sion? 

Ultimately,  the  question  devolves  to  developing  standard  labeling  analogous  to  the  nutritional 
labeling  standards  for  foodstuffs.  Backing  up  standard  labeling  will  be  the  strictures  of  “truth 
in  advertising”  and  the  legal  and  economic  consequences  of  violating  these  strictures. 
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5  Key  Results,  Open  Questions 


We  described  the  elements  of  a  PECT  and  their  relationships.  We  also  described  several  ways 
to  validate  a  PECT.  These  validations  are  essential  to  generate  trust  in  a  PECT  and  the  compo¬ 
nents  used  in  a  PECT.  We  also  described  a  number  of  concepts  that  are  pertinent  to  document¬ 
ing  and  packaging  predictable  assembly.  Ultimately,  we  hope  to  provide  guidelines  for  a 
standard  approach  to  labeling  components  and  prediction-enabled  component  technologies. 
We  have,  in  most  cases,  demonstrated  our  ideas  with  the  COMTEK-X  prototype. 

Not  surprisingly,  given  the  immature  state  of  this  research,  this  work  and  report  have  resulted 
in  a  number  of  questions: 

•  How  will  more  than  one  prediction  technology  be  integrated  in  PECT?  The  constraints 
placed  on  a  component  technology  by  two  prediction  technologies  may  be  incompatible  or 
may  interact  in  such  a  way  as  to  perturb  their  individual  or  combined  predictions. 

•  Assuming  that  more  than  one  prediction  technology  can  co-exist,  can  they  be  based  in  rad¬ 
ically  different  theories?  For  example,  can  one  theory  rest  in  formal  verification  while 
another  rests  in  a  more  empirical  approach?  Will  the  “seams”  between  these  be  visible? 

•  The  PECT  exemplar  described  in  this  report  emphasized  empirical  measurement  of 
resource  consumption — time,  in  this  case.  This  was  well  suited  to  empirical  validation. 
How  are  non-resource  attributes  such  as  security  to  be  empirically  validated? 

•  What  effect  does  PECT  have  on  the  feasibility  of  industrial  certification  of  components? 
Would  the  same  certification  approach  work  for  both  resource  and  non-resource  related 
component  properties? 

•  At  what  time  should  component  properties  be  evaluated  and,  possibly,  certified?  Proper¬ 
ties  can  be  evaluated  in  a  component  vendor  environment,  third-party  environment, 
deployment  environment,  assembly  environment,  and  end-execution  environment. 

These  and  many  other  questions  will  be  addressed  in  upcoming  research. 
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Appendix  A  Theoretical  Validity 


In  the  following,  A.X  and  C.X  are  sometimes  used  as  shorthand  for  A.latency  and  C.latency, 
respectively.  Also,  the  terms  ‘assembly’  and  ‘component’  refer  to  analytic  assemblies  and 
components  unless  otherwise  noted. 


A.1  Association  Validity — Derivation  of  A^^ 

Definitions 

The  following  definitions  are  used  throughout  the  derivation. 

1.  Component  C:  The  scheduled^  entity,  C. 

2.  Assembly  A:  The  set  of  components  A  =  {Cj,  C2,  -Cij}. 

3.  Schedule;  The  execution  order  of  components  in  A. 

4.  Cycle  A",  C*™:  The  n*  execution  of  the  schedule.  C™*  denotes  that  a  component  has  exe¬ 
cuted  m  times;  its  last  execution  was  in  the  n***  cycle,  denoted  A". 

5.  Component  latency  C.X:  The  duration  of  time  that  begins  when  the  COMTEK-A.  execu¬ 
tive  calls  the  execute  ( )  method  on  a  component,  and  ends  when  the  component  returns 
from  the  execute  ( )  method. 

6.  Assembly  latency  A.X:  The  latency  of  the  assembly  A,  defined  as  the  duration  of  time  that 
begins  when  the  COMTEK-X  executive  calls  Co".execute  ( )  on  an  arbitrary  first  com¬ 
ponent  Co  (i.e.,  VCj  e  A,  n>0  •  Cj"'^  a  -iCj"),  and  the  time  Cj^"  returns  from  its  Cij".exe- 
cute  ( ) ,  where  is  last  component  in  A"  (i.e.,  n>0,  Ck"'*  a  VCj  e  A  |  Cj  •  Cj"). 

7.  Steady  state:  An  assembly  is  said  to  be  operating  in  steady  state,  when  AKX  =  for 

all  j  >  k,  where  A*'  is  the  first  steady  state  cycle.  Prior  to  A*',  an  assembly  is  in  the  initializ¬ 
ing  state.  All  assemblies  eventually  reach  and  remain  in  steady  state.  (This  is  not  proven.) 


1 .  The  schedule-ah/e  entity  is.  in  COMTEK-X.  a  dynamic  link  library  (DLL)  that  conforms  to  the  COMTEK-X  API.  The  schedule- 
ed  entity  is  an  instance  of  the  schedule-able  entity  In  a  particular  assembly. 
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Observations 

8.  C.X  has  two  constituents: '  the  duration  of  time,  c,  that  C  executes,  and  the  duration  of 
time,  b,  that  C  waits  (“blocks”)  for  resources  that  are  required  to  complete  its  execution. 
That  is,X  =  e  +  b. 

9.  Different  COMTEK-A,  components  can,  and  usually  do,  perform  different  functions,  but 
each  component  must  execute  the  same  function  in  each  cycle  A".  Further,  the  function 
must  be  independent  of  its  data.  Therefore,  C.A  can  be  represented  by  a  single  time  dura¬ 
tion  rather  than  a  vector  of  durations,  and  e  is  constant  mX  =  e  +  b.  This  is  constraint  1) 
on  page  20.  This  constraint  greatly  simplifies 

10.  COMTEK-A  uses  round-robin  scheduling,  which  ensures  that  each  component  in  A  is  exe¬ 
cuted  exactly  once  per  cycle.  The  schedule  is  determined  by  data  dependencies  derived 
from  an  analysis  of  pipeline  connections  among  components.  The  subassembly  Cj  •  Cj^, 
where  denotes  an  allowed  interaction,  or  pipe,  is  interpreted  by  the  scheduler  as  assert¬ 
ing  that  Ck  depends  upon  data  produced  by  Cj.  For  Cj  •  C^,  the  scheduler  guarantees  that 
Cj".execute  ( )  executes  prior  to  C]^".execute  ( )  in  A". 

11.  From  10,  the  COMTEK-A  scheduler  guarantees  that  b  =  OinX  =  e  +  bjor  all  components 
that  depend  only  upon  other  components. 

12.  Components  may  also  exhibit  dependencies  on  non-component  resources,  that  is, 
resources  external  to  A.  These  dependencies  are  opaque  to  COMTEK-A:  the  scheduler 
does  not  take  these  external  dependencies  into  consideration  in  scheduling,  and 
COMTEK-A  makes  no  guarantees  about  bmX  =  e  +  b  when  blocking  on  external 
resources. 

13.  From  (11)  and  (12),  we  adduce  two  classes  of  component:  one  class  (0)  consists  of  those 
components  that  depend  only  upon  other  components,  the  other  (O)  consists  of  those  com- 
ponents  that  also  exhibit  external  dependencies.  That  is; 

a.  ©.A  =  e 

b.  4>.A  =  e  +  b,  where  e  is  a  known  constant,  and  &  is  a  variable  that  depends  upon  the 
interaction  protocol  with  the  external  resource,  e.g.,  blocking  or  non-blocking. 

14.  b  must  be  bounded  in  d>.A.  Since  the  COMTEK  component  technology  is  silent  on  exter¬ 
nal  dependencies,  we  must  impose  design  constraints  external  to  COMTEK  that  bound 
b  for  all  external  dependencies.  This  is  constraint  2)  on  page  20.  Again,  this  constraint 
greatly  simplifies  A/^. 


1 .  This  definition  is  arguably  true  of  latency  in  general. 

2.  As  a  mnemonic, 0  denotes  pipes  that  extend  beyond  an  assembly  boundary,  while  0  denotes  pipes  that  are  wholly  contained 
by  an  assembly  boundary. 
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a.  We  restrict  <1>  to  dependencies  on  periodic  events  x,  with  a  constant  period,  x.p.  Both  x 
and  x.p  are  independent  of  COMTEK-A,.  In  the  following,  we  usually  care  about  the 
period  and  not  the  event,  and  so  we  will  usually  denote  x.p  as  simply  p. 

b.  In  steady  state,  for  an  assembly  consisting  of  exactly  one  component  =  e  +  b 

=  e  +  (p-  e).  That  is,  the  amount  of  time  Ojj  blocks  on  x  (i.e.,  will  be  the  period 

x.p  less  the  amount  of  time  spent  executing  (i.e.,  O^.e). 

Derivation 

Ay\  predicts  A. A  in  steady  state.  We  derive  latency  prediction  functions  for  three  classes  of 

assemblies: 

•  f®  for  A  consisting  exclusively  of  ©  components. 

•  f^  for  A  consisting  exclusively  of  <I>  components. 

•  f®^  for  A  consisting  of  arbitrary  mixes  of  ©  and 


Each  are  derived,  in  turn. 


Class  1 :  O  Assemblies  (f®) 

15.  Base  case:  if  A  =  {©j},  then  A.A  =  ©j.e,  by  (5)  and  (13.a). 

16.  If  A  =  {©j,  ©jj},  then  A.A  =  ©j.e  +  ©j^.^,  where  ‘+’  is  arithmetic  addition. 

a.  Intuition:  Assume  COMTEK-A  is  infinitely  fast  so  that  scheduling  overhead  can  be 
ignored.  Since  ©  has  by  definition  no  blocking  time,  assembly  latency  will  consist 
only  of  component  execution  time. 

b.  Proof  Scheme*:  By  (6),  A.  A  is  the  time  elapsed  between  invocation  of  the  execute 
method  of  the  first  component  in  an  assembly,  and  the  return  from  the  execute  method 
of  the  last  component  in  that  assembly.  Figure  9  shows  the  execution  timeline  of  an 


1.  The  vertical  axis  represents  components,  while  the  horizontal  axis  represents  time.  Time  is  divided  into  ticks.  The 
parenthetical  subscript  represents  the  execution  duration  x  clock  ticks  for  component  © .  The  subscript  y)  represents 
the  execution  duration  x  and  period  of  external  resource  y  for  component  4>. 
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assembly  with  two  ©  components.  As  can  be  seen,  the  latency  of  the  assembly  is  the 
sum  of  the  latencies  of  the  two  components. 


I 
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I  I 

1  1  11 
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A.X  =  5 

H 

time 

Figure  9:  Execution  Timeline  for  an  Assembly  with  Two  0  Components 

17.  The  previous  case  can  be  extended  to  an  assembly  with  any  number  of  components.  If  A  = 
{©0,  ©1, ...  ©j},  then  A.X  =  f®(A)  =  ©q-c  +  ©|.e  +  ...  +  ©j.e  =  ^  ©j.e 

©j€  A 

Class  2:  O  Assemblies  (f*^) 

18.  Base  case:  if  A  =  {Oj},  then  A.X  =  max(<l)j.e,  Oj.p),  where  max  is  the  larger  value  of  Oj.e 
and  Oj.p. 

a.  Intuition:  We  might  expect  Oj.e  <  d>j.p,  but  this  need  not  be  so.  It  is  true  that  if  Oj.e  > 

then  will  miss  at  least  one  periodic  event  x.  The  significance  of  this,  however, 

will  depend  upon  the  semantics  of  the  application.* 

b.  Proof  Scheme:  Figure  10  shows  the  timeline  for  the  three  possible  cases:  where  Oj.e  < 
Oj.p,  where  3>j.e  =  Oj.p,  and  where  <l>j.e  >  Oj.p. 


1 .  In  some  applications,  for  example  audio  playback,  all  external  events  must  be  met.  In  other  applications,  for  example  radar 
position,  only  the  “last"  event  must  be  met;  in  this  situation,  we  may  only  need  to  put  a  bound  on  the  number  of  periodic 
events  missed  per  unit  of  time. 
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Figure  10:  Timeline  for  O  Components  with  e<p,  e=p,  and  e<  p 


19.  If  A  =  {Oj,  ^1^},  then  A.X  =  max(  (3>j.e  +  ),  max(  (pyp,  ) 

a.  Case  1,  where  Oj.p  =  <5|j.p.-  A.X  =  max(  (Oj.e  +  <E>jj.e ),  p) .  Figure  11  depicts  this  sit 
uation. 


0;.e  +  <t>v.e 


Figure  11:  Timeline  for  Two  Components  with  the  Same  Period 
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b.  Case  2,  where  0j.p  This  case  can  be  considered  as  a  special  case  of  Case  1, 

with  the  term  p  in  Case  1  replaced  by  max(Oj.p,  O^.p).  Figure  12  shows  how  the  larg¬ 
est  period  dominates  the  latency  of  the  assembly. 
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Figure  12:  Timeline  for  Two  4>  Components  with  Different  Periods 


20.  If  A=  {Oq,  O,.  ...  «I)j},then 
A.^=  AA) 

=  max(  (^Q.e  +  <^].e  +  ...  +  Oj.e  ),  max  (Oo.p,  <I>i.p,...,<l>j.p) ) 

=  max(  ^  maxIIOj  e  A|  CJj.p})). 

4>j  e  A 

Class  3:  ©O  Assemblies  {i®^) 

21.  We  observe  that  if  e  >p  in  some  0'.(e,  p),  then  O'  will  never  block  on  the  periodic  event 
x.p.  A  O  component  that  never  blocks  is,  for  the  purpose  of  latency  computation,  indistin¬ 
guishable  from  a  0  component  whose  latency  is  Q.e.  We  can  also  view  this  from  the  other 
direction,  and  in  a  more  specialized  form.  Specifically,  a  ©  component  is  indistinguishable 
from  a  O  component  with  O.e  =  ©.e  and  O.p  =  0. 

22.  Using  (21),  we  can  apply  (20)  to  the  general  case  where  A  consists  of  a  set  of  ©  and  O 
components: 


/ 

y 

A 

\ 

A.  latency  =  f(A)  =  max 

S  ej ' 

,max({<Dje  A  |  Oj.p}) 

V 

e  A 

e,€  A  ^ 

Eq.  5 


where  the  term  Z©j.e  reflects  the  relation  ©j.e  =  O.fe,  0).  Naturally,  since  p  =  0  for  the  © 
components,  we  need  not  introduce  an  additional  term  into  the  set  of  component  periods. 
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23.  Although  it  may  be  useful  for  other  purposes  to  distinguish  between  ©  and  O  components, 
this  distinction  can  be  represented  implicitly  using  (21).  That  is,  we  might  dispense  with 
the  distinction  between  0  and  O.  If  all  components  C  possess  two  attributes,  C.e  execution 
time,  and  C.p  the  period  of  an  external  dependency,  then  ©  components  would  be  repre¬ 
sented  implicitly  by  letting  C.p  =  0. 

24.  Under  the  convention  in  (23),  Eq.  5  can  be  simplified  as  follows: 


A.  latency  =  f(A)  =  max 


X 

^  Cj.e,  max({Cje  A  |  Cj.p}) 

'"Cj  6  A  ^ 


Eq.6 


A.2  Interpretation  Validity 

We  must  demonstrate  that  every  constructive  assembly  can  be  expressed  as  an  analytic  assem¬ 
bly  (completeness),  and  that  there  is  a  unique  latency  prediction  for  each  constructive  assem¬ 
bly  (consistency). 


A.2.1  Completeness 

Each  component  C  €  A  is  converted  to  an  analytic  component  of  one  of  two  types,  based  on 
these  rules: 

•  If  C  interacts  with  a  periodic  component  C’  external  to  A,  then  it  is  represented  by  an  ana¬ 
lytic  component  of  type  O,  where  the  latency  property  is  denoted  as  0.(e,  p),  where  ‘e’ 
denotes  the  execution  time  of  C  and  ‘p’  denotes  the  period  of  C’. 

•  If  C  interacts  only  with  other  components  in  A,  then  it  is  represented  by  an  analytic  com¬ 
ponent  of  type  ©,  where  the  latency  property  is  denoted  as  ©.e,  where  ‘e’  denotes  the  exe¬ 
cution  time  of  C. 

Since  interactions  in  A  (that  is,  pipes)  are  ignored,  we  need  not  consider  ordering.  We  there¬ 
fore  treat  A  as  the  set  of  components  {a^,  02,  03, nij}.  So  constructed,  it  is  clear  that  every 
possible  COMTEK-^  constructive  assembly  can  be  represented  as  an  analytic  assembly  in  Ay^. 

The  analytic  assembly  can  also  be  ordered  using  the  operator  *•’  :r  x  F  — >  F,  where  F  =  O  L.1  ©. 
Informally,  *•’  is  a  left-associative  concatenation  operator  that  expresses  the  temporal  relation, 
‘before’.  That  is,  •  02  means  ‘aj  executes  before  02’-  In  this  case,  the  analytic  assembly 
would  be  represented  as  the  string  A^  =  •  ^2  *  ^3  •  •••  *  for  j  >  1,  where  aj  ^2,  ^3  —  ^  T- 

This  temporal  ordering  would  be  appropriate  and  necessary  should  A^  be  extended  to  address 
non-steady  state  latency. 
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A.2.2  Consistency 

Since  Eq.  5  and  Eq.  6  involve  operations  (1  and  max)  that  do  not  depend  on  the  order  of  their 
arguments,  all  analytic  assemblies  produced  using  the  interpretation  in  Section  A. 2.1  will  be 
mathematically  equivalent,  regardless  of  the  order  in  which  components  are  considered. 

QED 
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Appendix  B  Empirical  Validity 


Empirical  validity  consists  in  quantifying  the  accuracy  and  repeatability  of  predictions  made 
using  an  analytic  model  by  statistically  comparing  those  predictions  with  actual  measurements 
of  assembly  properties.  The  process  of  empirically  validating  a  PECT  can  be  summarized  in 
the  following  steps: 

1.  Measure  components. 

2.  Design  assemblies  with  the  components  and  predict  the  property  of  interest. 

3.  Build  the  assemblies  and  measure  the  property  of  interest. 

4.  Statistically  analyze  the  results. 

To  empirically  validate  COMTEK-A,  we  selected  the  Audio  Playback  (APB)  application.  This 
simple  application  was  conducive  to  studying  end-to-end  latency,  the  property  that  we  wanted 
to  predict.  APB  is  comprised  of  the  following  components: 

•  Audio  Input — treads  digital  audio  data  from  devices  within  the  host  operating  system  and 
makes  that  data  available  to  other  components.  It  has  two  outputs,  left  and  right  audio 
channels. 

•  Generator — generates  a  digital  audio  signal  (frequency  and  amplitude).  It  has  one  output. 

•  Adder — ^produces  a  digital  audio  signal  from  two  input  signals  (additive).  It  has  two  inputs 
and  one  output. 

•  Subtracter — produces  a  digital  audio  signal  from  two  input  signals  (subtractive).  It  has 
two  inputs  and  one  output. 

•  Audio  Out — outputs  digital  audio  data  to  devices  within  the  host  operating  system  to  pro¬ 
duce  sound.  It  has  two  inputs,  left  and  right  audio  channels. 

•  Audio  View — outputs  a  graphical  representation  of  input  digital  audio  data.  It  has  one 
input. 

The  following  sections  describe  how  the  experiments  to  empirically  validate  COMTEK-X 
were  conducted. 
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B.1  Measuring  the  Latency  of  Individual  Components 


Latency  for  an  individual  component  is  defined  as  the  duration  of  time  starting  from  the  invo¬ 
cation  of  its  execute  ( )  method  to  its  return  from  that  method  (see  Figure  13).  The  execute 
method  is  required  by  COMTEK-X;  it  is  the  method  in  which  the  component  performs  its 
work. 
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Figure  13:  Latency  of  a  COMTEK-X  Component 

Measuring  the  latency  of  an  individual  component  was  not  straightforward.  First,  a 
COMTEK-A  assembly  cannot  execute  until  it  is  “fully  connected”  (see  Figure  7  on  page  17). 
Which  means,  in  short,  that  components  in  a  COMTEK-X  application  having  required  inputs 
(‘InputPort’)  must  be  connected  to  a  source  (‘OutputPort’)  before  the  COMTEK-X  executive 
will  run  the  application.  A  COMTEK-X  component  can  have  zero  or  more  InputPorts  and  zero 
or  more  OutputPorts,  but  must  have  at  least  one  port;  it  may  not  have  zero  of  both.  Therefore, 
we  would  have  to  have  at  least  two  components  in  a  COMTEK-X  application  to  calculate  the 
latency  of  the  component  we  wanted  to  measure.  Lastly,  we  wanted  to  measure  the  latency  of 
a  component  in  an  external  environment  to  better  simulate  the  role  of  a  component  property 
certifier. 

For  these  reasons,  we  created  a  test  harness  to  measure  the  latency  of  single  components.  The 
test  harness  supplied  the  services  of  COMTEK-X  sufficient  to  execute  the  component,  and  no 
more.  A  useful  analogy  is  an  engine  placed  in  a  dynamometer  to  measure  its  performance.  To 
run  the  test,  we  put  oil  in  the  engine,  hook  up  a  battery  and  fuel  line  (input),  attach  it  to  a  wir¬ 
ing  harness  (for  control),  hook  up  a  transmission  and  exhaust  (output),  and  conduct  the  perfor¬ 
mance  test.  This  is  essentially  what  the  COMTEK-X  test  harness  does  for  COMTEK-X 
components. 

The  test  harness,  shown  in  Figure  14,  supported  loading  and  executing  one  COMTEK-X  com¬ 
ponent  at  a  time.  A  component  in  the  test  harness  could  be  executed  any  number  of  times 
while  the  test  harness  measured  and  recorded  the  time  the  component  spent  in  its  execute  ( ) 
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method.  The  test  harness  computed  several  parameters  for  a  component  depending  on  its  set¬ 
tings,  but  the  most  useful  were  the  average  execution  time  and  the  standard  deviation  of  the 
execution  times. 


iiCOMTEK-lambda  Test  Harness 


Execution  Runs; 


2000 


elapsed  lime 
2.1 3051 4350  seconds 


average  execulion  lime 
0.000544001  seconds 

minimum  execution  time 
0.000540517  seconds 

maximum  execulion  lime 
0.00118780G  secbnds 


Latency  (ms): 


number  of  executions 


execution  statistics 


execution  time  std.  dev. 
0.000019703 


LoadLibrary  '  I  Execute  UnLoadLibrary 


delay  before  executing 


execution  options 

./ 

Reset  Inputs  | 

I”  Save  statistics 
I”  Detailed  Timing 
!”  Capture  Timing 
T~  Capture  Output(s) 
1“  Meiers 
jn  Debug  ' 


Figure  14:  COMTEK-X  Component  Test  Harness 

Input  and  output  (I/O)  from  a  component  in  the  test  harness  were  handled  in  precisely  the 
same  manner  as  I/O  within  the  COMTEK-X  run-time  environment,  as  memory  streams. 
Because  the  test  harness  mimicked  the  precise  behavior  of  the  memory  input  and  output 
streams,  we  not  only  established  a  suitable  environment  for  testing  components  (with  actual 
test  data),  we  also  achieved  a  very  close  approximation  of  the  individual  component  latencies 
we  would  expect  in  the  actual  COMTEK-X  runtime. 

Measuring  the  time  spent  in  the  execute  ( )  method  gives  the  latency  of  a  ©  component,  but 
for  0  components  we  also  need  the  period  of  the  event  that  <I>  components  depend  on.  To  mea¬ 
sure  the  actual  execution  time  of  a  O  component,  exclusive  of  any  blocking,  we  inserted  a 
timed  delay  that  was  longer  than  the  external  period  before  dispatching  the  execute  ( ) .  In 
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this  way,  only  the  time  spent  performing  work  was  measured,  not  time  waiting  for  the  period 
to  expire.  The  length  of  this  timed  delay  can  be  set  in  the  test  harness,  as  can  be  seen  in  Figure 
14. 

In  addition  to  the  components  that  were  used  in  the  original  APB  application,  we  created  vari¬ 
ations  of  two  of  them  to  have  a  broader  palette.  One  variation  was  a  delayed  adder  that  per¬ 
formed  the  same  operation  as  the  original  adder  but  also  had  a  forced  delay  that  would  make 
its  execution  take  roughly  60ms.  The  other  variation  was  a  O  version  of  the  subtractor  compo¬ 
nent.  This  new  subtractor  would  depend  on  a  periodic  event  of  80ms  in  addition  to  its  20ms 
required  to  perform  its  computation. 

The  COMTEK-A,  framework  is  implemented  on  the  Microsoft  Windows^’"*  platforms.  We  ran 
the  empirical  validation  experiments  in  one  workstation  running  Windows  2000  Professional 
operating  on  a  DELL  Dimension  4100  comprised  of  a  Pentium^'^  III  IGHz  processor,  512Mb 
of  system  RAM,  ATI  Rage  Pro  graphics  adapter  with  16Mb  RAM,  and  a  Creative  Sound 
Blaster  AudioPCI  audio  device.  To  minimize  the  effects  of  other  processes  on  the  measure¬ 
ments,  the  test  harness  sets  itself  to  high  priority  while  running  the  benchmarks. 

Table  2  shows  the  results  of  the  measurements  for  all  the  components.  What  we  consider  to  be 
the  certified  latency  of  a  component  is  the  average  of  a  large  sample  of  measurements.  In  our 
empirical  validation,  we  took  20,000  measurements  for  each  component.  We  have  included  in 
Table  2  the  maximum  latency  estimate  error  at  95%  confidence.  For  example,  for  the  compo¬ 
nent  Generator,  we  can  be  95%  confident  that  the  average  of  the  sample  (i.e.,  what  we  use  as 
latency)  is  off  by  less  than  9.61655E-07  from  the  true  mean  of  the  latency. 


Table  2:  Latency  of  Individual  Components^ 


Component 

Execution  (e) 

(in  seconds) 

Period  (p) 

(in  seconds) 

Average 

Std.  Dev. 

Max.  error 

Average 

Std.  Dev. 

Max.  error 

Audio  Input 

0.000161557 

0.000371361 

5.1468E-06 

0.046409895 

0.002502847 

3.46877E-05 

Generator 

0.000250372 

0.000069387 

9.61655E-07 

Adder 

0.000030612 

0.000037586 

5.20915E-07 

Subtractor 

0.000030488 

0.000037678 

5.2219E-07 

Audio  Out 

0.000185987 

0.000050028 

6.93353E-07 

0.046400214 

0.009928901 

0.000137608 

Audio  View 

0.001159573 

0.000206997 

2.86883E-06 

Delayed  Adder 

0.060077160 

0.000370087 

5.12914E-06 

Dependent  Subtractor 

0.020026375 

0.000174137 

2.41342E-06 

0.080112472 

0.000530736 

7.35563E-06 

a.  The  latencies  shown  In  the  table  are  the  average  of  20,000  measurements  taken  for  each  component. 
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B.2  Predicting  the  Latency  of  an  Assembly 

Having  measured  all  the  components  we  were  going  to  use,  we  were  ready  to  predict  the 
latency  of  the  different  assemblies  that  would  constitute  our  empirical  validation.  In  addition 
to  the  APB  application  shown  in  Figure  15,  we  created  eight  more  assemblies.  Figure  15 
shows  one  APB  assembly  as  it  is  displayed  in  the  COMTEK-X  assembly  environment. 


Figure  15:  Audio  Playback  Application  Assembly 


For  each  assembly,  we  calculated  the  predicted  latency  using  the  analytic  model  (Eq.  6  on 
page  37): 


A. latency  =  f(A)  =  max 


Cj.e,max({Cje  A  I  Cj.p}) 


^Cj€  A 


Eq.6 
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Table  3  shows  the  components  used  in  the  APB  assembly  with  their  analytic  interfaces.  In  the 
table,  we  have  highlighted  the  sum  of  the  execution  times,  and  the  largest  period. 


Table  3:  Components  Used  in  the  APB  Assembly 


Component 

Execution  (e) 
(in  seconds) 

Period  (p) 

(in  seconds) 

Audio  Input 

0.000161557 

0.046409895 

Generator  1 

0.000250372 

Generator  2 

0.000250372 

Adder 

0.000030612 

Subtractor 

0.000030488 

Audio  Out 

0.000185987 

0.046400214 

Audio  View 

0.001159573 

Sum 

0.002068961 

Filling  in  the  values  in  the  previous  formula,  we  obtained  the  predicted  assembly  latency: 
A. latency  =  max (0.00206896,  0.0464099)=  0.0464099 


Following  the  same  procedure  for  all  the  assemblies  in  our  experiment,  we  obtained  the  pre¬ 
dicted  latencies  shown  in  Table  4.  The  rest  of  the  assemblies  can  be  found  in  Appendix  C. 


B.3  Measuring  the  Latency  of  an  Assembly 

As  described  earlier,  the  COMTEK-X  framework  pre-computes  the  schedule  (order  of  execu¬ 
tion)  of  components  for  a  COMTEK-X  assembly  at  assembly  time.  This  schedule  is  simply  a 
fixed  list;  component  X  goes  first,  then  Y,  and  so  on  until  the  last  component  is  dispatched  at 
which  time  the  scheduler  reverts  back  to  the  first  component  in  the  ordered  list  and  continues 
in  a  cyclic  fashion.  We  defined  assembly  latency  as  the  elapsed  time  between  two  consecutive 
invocations  of  the  exscute  ( )  method  of  some  component  Cj;  S  A. 

There  is  a  slight  difference  between  this  definition  of  measured  assembly  latency  and  the 
assembly  latency  as  defined  in  the  analytic  model.  Figure  16  depicts  this  difference.  Since  the 
COMTEK-X  assembly  environment  pre-computes  the  schedule,  there  is  little  or  no  scheduling 
overhead  at  run  time.  Therefore,  we  assume  A  to  be  negligible. 
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In  addition  to  instrumenting  the  COMTEK-A  run-time  environment  to  measure  the  cycle  time 
of  an  assembly,  we  also  instrumented  it  to  compute  and  record  the  total  number  of  cycles  actu¬ 
ally  executed  (i.e.,  number  of  samples  taken)  along  with  average  and  standard  deviation  of  the 
cycle  times.  For  both  the  test  harness  and  the  COMTEK-A,  executive,  the  overhead  of  instru¬ 
menting  and  calculating  component  latency  was  taken  into  consideration  when  measuring  A.X 
and  C.A.  Table  4  shows  the  measured  latency  for  all  the  assemblies  in  the  experiment. 


Table  4:  Predicted  and  Measured  Latency  for  the  Assemblies  in  the  Experiment 


Assembly 

Predicted 

Latency 

Measured  Latency 

Average 

Samples 

Std.  Dev. 

1 

0.046409895 

0.046432115 

15000 

0.005848142 

2 

0.080451079 

0.080130336 

15000 

0.000491377 

3 

0.141687812 

0.140204650 

15000 

0.000603960 

4 

0.060675076 

0.060083619 

_  _  i 

15000 

0.000278201 

5 

0.046400214 

0.046401125 

15000 

0.009964861 

6 

0.046400214 

0.046404150 

15000 

0.035717654 

7 

0.003131486 

0.003206257 

15000 

0.000310752 

8 

0.046400214 

0.046401195 

15000 

0.009883773 

9 

0.001440557 

0.001442853 

15000 

0.000242908 
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B.4  Statistical  Analysis  of  the  Results 


Although  just  by  looking  at  the  predicted  and  measured  latencies  in  Table  4  we  could  see  that 
the  analytic  model  produced  useful  predictions,  we  needed  to  establish  credibility  in  the  model 
by  backing  it  with  statistical  evidence.  We  used  the  magnitude  of  relative  error  (MRE)  and 
correlation,  two  methods  that  have  been  used  in  the  empirical  validation  of  other  software 
related  estimation  models  [Kemerer+87]. 


MRE  is  a  measure  of  the  percentage  error  of  the  predicted  latency,  and  it  is  defined  as  follows: 


MRE= 


A. 7  -  A.X' 
A.X' 


Eq.7 


where  A.X  is  the  predicted  latency  and  A.X'  is  the  measured  latency.  If  we  used  absolute  error, 
it  would  be  difficult  to  compare  the  errors  for  different  predictions.  For  example,  an  error  of 
0.3  ms  would  be  relatively  small  for  a  latency  of  140  ms  while  it  would  be  very  large  for  a 
latency  of  1.44  ms.  Therefore,  we  want  to  use  relative  error  instead  of  absolute  error  so  that  we 
can  compare  and  analyze  the  predictions  for  different  assemblies  together.  In  addition,  using 
the  absolute  value  of  the  relative  error  prevents  errors  with  opposite  signs  to  cancel  out  when 
the  average  of  them  is  taken.  Table  5  shows  the  MRE  for  all  sample  predictions. 

Table  5:  MRE  for  the  Predicted  Latency 


Assembly 

MRE 

log  MRE 

1 

0.000479 

-3.320074412 

2 

0.004003 

-2.397639777 

3 

0.010579 

-1.975573828 

4 

0.009844 

-2.006832907 

5 

0.000020 

-4.707010133 

6 

0.000085 

-4.071501732 

7 

0.023320 

-1.632265142 

8 

0.000021 

-4.674860158 

9 

0.001591 

-2.798250203 

Average 

0,005549 

-3.064890 

For  the  nine  assemblies  in  our  experiment,  the  average  magnitude  of  relative  error  was  about 
0.5%.  To  infer  how  the  model  would  behave  when  used  in  other  situations,  we  wanted  to 
define  a  confidence  interval  for  the  expected  value  of  MRE.  In  other  words,  we  wanted  to  cal¬ 
culate  a  range  within  which  the  mean  of  all  the  MREs  is  most  likely  to  fall.  The  statistical 
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methods  to  do  this  required  either  that  data  be  normally  distributed  or  that  the  sample  size  be 
greater  than  or  equal  to  30.  We  only  had  a  sample  size  of  9,  so  we  checked  whether  the  data 
was  normally  distributed  or  not. 

We  used  the  Shapiro-Wilk  test  for  normality.  The  results  were  W  =  0.7621  and  p-value  = 
0.007482,  where  W  is  the  statistic  used  in  the  test.  The  p-value  is  used  to  decide  whether  to 
reject  or  accept  the  null  hypothesis  that  the  data  is  normally  distributed.  When  the  p-value  is 
less  than  or  equal  to  the  level  of  significance  used,  then  the  null  hypothesis  of  normality  is 
rejected.  Using  a  level  of  significance'  a  =  0.05,  we  concluded  that  the  sample  data  was  not 
normally  distributed. 

We  applied  a  log  transformation  to  the  sample  data  intending  to  make  it  normally  distributed. 
The  values  of  the  transformation  are  also  shown  in  Table  5.  We  again  used  the  Shapiro-Wilk 
test  on  the  transformed  values,  with  the  result  that  W  =  0.9  and  p-value  =  0.2522.  Since  the  p- 
value  was  greater  than  the  level  of  significance,  we  accepted  the  hypothesis  of  the  transformed 
data  being  normally  distributed.  With  this,  we  defined  a  95%  confidence  interval  for  the 
expected  value  of  the  transformed  MRE; 

-3.976083  <  E(logMRE)  <  -2.1537 


Applying  anti-logarithm  to  revert  the  transformation,  we  obtained  the  confidence  interval  for 
the  MRE: 


0.000106  <  E(MRE)  <  0.007019 

This  means  that  there  is  a  probability  of  95%  that  the  average  MRE  for  all  the  predictions  (i.e., 
the  population  of  the  predictions)  will  fall  within  this  interval.  To  be  clear,  this  does  not  mean 
that  every  single  MRE  value  will  fall  within  this  range;  we  are  estimating  the  average. 

The  second  method  we  used  for  evaluating  the  empirical  validity  of  this  PECT  was  the  corre¬ 
lation.  We  used  the  predicted  latency  as  the  independent  variable,  and  the  actual  latency  as  the 
dependent  variable.  We  obtained  a  sample  coefficient  of  determination  R  =  0.999965721, 
meaning  that  there  is  an  almost  perfect  correlation  between  the  predictions  and  the  actual 
latencies  of  the  assemblies.  In  other  words,  we  can  say  that  the  analytic  model  accounts  for 
99.99%  of  the  variations  in  the  actual  latency.  Since  the  sample  size  we  were  using  was  not  too 
large,  we  wanted  to  be  sure  that  this  correlation  could  not  have  happened  by  chance.  The  criti¬ 
cal  value  of  R  for  7  degrees  of  freedom  (N-2)  at  a  1%  significance  level  (a  =  0.01)  is  0.798. 
The  correlation  coefficient  for  our  sample  is  /?  =  0.99998286,  which  is  greater  than  the  critical 
value.  Therefore,  we  conclude  that  the  correlation  is  statistically  significant. 


1 .  The  significance  level  a  is  the  probability  of  rejecting  the  null  hypothesis  when  it  is  true. 
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In  light  of  these  results,  we  believe  that  the  association  between  the  constructive  and  analytic 
models  in  our  exemplar  PECT  is  valid.  In  addition,  the  fact  that  the  analytic  model  explains  or 
predicts  99.99%  of  the  latency  and  that  its  predictions  have  a  relative  error  less  than  1%  on 
average  shows  strong  evidence  that  the  predictions  made  by  the  analytic  model  are  accurate. 
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Appendix  C  Assemblies  Used  for 

Empirical  Validation 


The  following  diagrams  show  the  nine  assemblies  that  were  used  for  the  empirical  validation 
ofthisPECT. 


Figure  17:  Assembly  1 


Figure  18:  Assembly  2 
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Audio  Input 


Figure  22:  Assembly  6 


Figure 23:  Assembly/ 
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Figure  24:  Assembly  8 


Figure  25:  Assembly  9 
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